Papers with synthetic task
Mitigating Copy Bias in In-Context Learning through Neuron Pruning (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models have shown impressive few-shot in-context learning abilities, but they are prone to a ‘copying bias’, where they copy answers from provided examples instead of learning the underlying patterns. |
| Approach: | They propose a method to prune neurons that prioritize copying over generalization and adopt a task-recognition perspective on ICL and examine task vectors induced by the model. |
| Outcome: | The proposed method improves performance across a diverse set of ICL tasks while maintaining or improving the model’s general capabilities. |
Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change (P19-1)
Copied to clipboard
| Challenge: | State-of-the-art lexical semantic change detection models suffer from noise stemming from vector space alignment. |
| Approach: | They propose a method to simulate lexical semantic change and control for possible biases by avoiding alignment. |
| Outcome: | The proposed method outperforms state-of-the-art models on a synthetic task and a manual testset. |
Deep Neural Models of Semantic Shift (N18-1)
Copied to clipboard
| Challenge: | Diachronic distributional models track changes in word use over time using a continuous variable and a synthetic task to measure the semantic trajectory of a word. |
| Approach: | They propose a deep neural network diachronic distributional model that represents time as a continuous variable and model a word’s usage as . a synthetic task which measures how well a model captures the semantic trajectory of a . word over time. |
| Outcome: | The proposed model can capture the semantic trajectory of a word over time and can measure the speed of lexical change. |
mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models? (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained multilingual models exhibit cross-lingual transfer ability, which is often attributed to a learned language-neutral representation during pretraining. |
| Approach: | They propose a synthetic task, Multilingual Othello, as a testbed to investigate the factors that contribute to the learning of a language-neutral representation. |
| Outcome: | The proposed approach induces the learning of language-neutral representation and facilitates cross-lingual transfer. |
HistAlign: Improving Context Dependency in Language Generation by Aligning with History (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language models (LMs) can generate hallucinations and incoherent outputs due to their weak context dependency. |
| Approach: | They propose a training approach to ensure good cache alignment so that the model receives useful signals from the history. |
| Outcome: | The proposed approach improves text coherence and faithfulness on diverse language generation tasks. |
Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models (2023.emnlp-main)
Copied to clipboard
Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, Mrinmaya Sachan
| Challenge: | Recent work has shown that language models (LMs) have strong multi-step (i.e., procedural) reasoning capabilities. |
| Approach: | They propose a mechanistic interpretation of language models for multi-step reasoning tasks by introducing a new probing approach that recovers the reasoning tree from the model’s attention patterns. |
| Outcome: | The proposed model implicitly embeds a reasoning tree resembling the correct reasoning process within it, and detects the information from the model’s attention patterns for most examples. |
CoLAKE: Contextualized Language and Knowledge Embedding (2020.coling-main)
Copied to clipboard
| Challenge: | Existing models for integrating factual knowledge into pre-trained language models are shallow, static, and separately pre-train entities. |
| Approach: | They propose a method which integrates knowledge contexts from large-scale knowledge bases into a unified data structure. |
| Outcome: | The proposed model outperforms existing models on knowledge-driven tasks and knowledge probing tasks. |
Unleashing Multi-Hop Reasoning Potential in Large Language Models through Repetition of Misordered Context (2025.findings-naacl)
Copied to clipboard
| Challenge: | Multi-hop reasoning requires multi-step reasoning based on supporting documents within a given context. |
| Approach: | They propose a method that prompts the model by repeatedly presenting the context. |
| Outcome: | The proposed method improves the F1 score by 30%p on multi-hop QA tasks and increases accuracy by 70%p on a synthetic task. |